United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
I nilid Stall-, Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. CONFIRMATION NO. 



10/743.3 +3 



12/23/2003 



32864 7590 07/21/2009 

FISH & RICHARDSON, P.C. 
PO BOX 1022 

MINNEAPOLIS, MN 55440-1022 



AMSDELL, DANA 



PAPER NUMBER 



NOTIFICATION DATE | DELIVERY MODE 
07/21/2009 ELECTRONIC 



Please find below and/or attached an Office communication concerning this application or proceeding. 

The time period for reply, if any, is set in the attached communication. 

Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the 

following e-mail address(es): 

PATDOCTC@fr.com 



PTOL-90A (Rev. 04/07) 



l/ffflrC? nVrliUli Otfff Iff ids y 


Application No. 

10/743,343 


Applicant(s) 

RAIYANI ET AL. 


Examiner 

DANA AMSDELL 


Art Unit 

3627 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address — 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )KI Responsive to communication(s) filed on 04 May 2009 . 
2a )□ This action is FINAL. 2b)^ This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) |EI Claim(s) 58-64,68-77 and 81-96 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) |EI Claim(s) 58-64, 68-77 and 81-96 is/are rejected. 

7) 0 Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) Q The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

20 Certified copies of the priority documents have been received in Application No. . 

3.Q Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attach ment(s) 

1) ^| Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Paper No(s)/Mail Date. . 

3) Information Disclosure Statement(s) (PTO/SB/08) 5 ) □ Notice of Informal Patent Application 
Paper No(s)/Mail Date 6/15/2009 . 6) □ Other: . 



PTOL-T26 d (Rev e 08-06r 



Office Action Summary 



Part of Paper No./Mail Date 20090712 



Application/Control Number: 10/743,343 Page 2 

Art Unit: 3627 

DETAILED ACTION 

Response to Remarks 

1 . Examiner acknowledges cancellation of claims 65-67, 78-80, and 81 - 90 and 
amendments to independent claims 58, 71 , and 84 for the purpose of continued 
examination. 

Applicant's remarks regarding Bergman teaching an arena venue, thus not anticipating 
a warehouse environment, have been fully considered but they are not persuasive. In 
light of the Specifications, Examiner interprets 'warehouse environment' to be any 
venue that holds an inventory; and employs "worker[s] who often use their hands to 
perform tasks" (page 2, line 1). Claims 58, 71 and 84 as amended will be examined on 
their merit in the immediate action. 

Claim Rejections - 35 USC § 102 

2. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 1 02 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

3. Claims 58-64, 68-77, and 81-96 are rejected under 35 U.S.C. 102(b) as being 
anticipated by Bergman et al. (US 7174308), herein "Bergman". 
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4. Regarding claims 58, 71 and 84 are directed to multimodal warehouse 
applications, Bergman teaches: 

• one or more computers; and 

• a computer-readable medium coupled to the one or more computers having 
instructions stored therein (Fig. 7); which, when executed by the one or more 
computers, cause the one or more computers to perform operations comprising: 

providing a multi-modal user interface on a mobile device (Fig. 1, elm. 36), 
the multi-modal interface for performing inventory management tasks (column 5, 
lines 25-32) in an environment that holds an inventory (Fig. 5, elm. 100/102) and 
employs workers who often use their hands to perform tasks, henceforth 
'warehouse' (Fig. 1 , elms. 32/ 34), and the multi-modal interface enabling output 
and user input of information using both first and second modalities that 
collectively include a voice-based modality and a touch-based modality (Fig. 1, 
elms 20/22/24); 

receiving a first user input identifying a job type mode (column 5, lines 8-10 
-order), the first user input being input using either the first modality or the 
second modality as chosen by a user of the mobile device, and the job type 
mode being selected from among a stocking mode in which the user enters one 
or more items into a dispenser, henceforth 'bin', in the warehouse environment a 
picking mode in which the user removes the one or more items from the bin in 
the warehouse environment, and a counting mode in which the user counts the 
one or more items in the bin in the warehouse environment (column 6, lines 18- 
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18), 

transmitting information identifying the user-selected job type mode to an 
inventory management system a server, in a format consistent with the modality 
in which the first user input was received (column 4, lines 5-15), 

responsive to transmitting the information, receiving a response to the 
information identifying the user-selected job type mode from the inventory_ 
management system (Fig. 3, block 84; and column 4, lines 34-44), in formats 
consistent with both the first modality and the second modality (column 6, lines 
44-53), the response identifying a location at which the user will perform a 
function related to the user-selected job type mode on the one or more items, a 
description of the one or more items, and a best 'zone' (henceforth 'route') from a 
present location of the user to the bin an item (column 4, lines 41-50; and Fig. 
10-Zone; and column 9, lines 29-34), 

providing the response to the user using both audio and display 
components of the multi-modal user interface, in the formats consistent with both 
the first modality and the second modality (column 7, lines 30-50), 

receiving a second user input using either the first modality or the second 
modality as chosen by the user, the second user input indicating that the user 
has completed the selected function, and including count information or location 
information associated with the one or more items as observed by the user and, 
when the voice-based modality is used, a phonetically distinct checkword 
associated with the bin to indicate completion of the selected function (column 4, 
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lines 34-67-wherein end of order flag or indicator is a checkwork in voice-based 
modality, henceforth checkword), 

updating inventory data for the one or more items item based on the count 
information or the location information, the updated inventory data being 
accessible using the formats consistent with both the first modality and the 
second modality (column 5, lines 7-15), and 

transmitting information indicating that the user has completed the selected 
function related to the user-selected job type mode to the inventory_ 
management, in a format consistent with the modality in which the second user 
input was received (column 4, lines 36-67; and column 5, lines 6-7) 

5. Regarding claims 59, 72 and 85, Bergman teaches wherein the operations 
further comprise: 

outputting the updated inventory data in a format consistent with either the first modality 
or the second modality, such that the inventory data is maintained during performance 
of the inventory management tasks (Fig. 5, elm. 102; and column 5, lines 28-41). 

6. Regarding claims 60, 73 and 86, Bergman teaches wherein the job type mode 
comprises job data (Fig. 3, elm. 80-wherein an order is representative of a job' and 
order content represents job data). 
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7. Regarding claims 61 , 74 and 87, Bergman teaches wherein the multi-modal 
interface further enables output and user input information using first modality and the 
second modality collectively include two or more of a voice modality, a Radio Frequency 
Identification Device (RFID) modality, or a barcode modality (Fig. 7, elms. 151/152/153; 
and column 7, lines 30-50), 

8. Regarding claims 62-63, 75-76 and 88-89, Bergman teaches wherein the 
operations further comprise receiving a pick list identifying the one or more items and 
the function; and further displaying the pick list using a visual user interface, and 
outputting, using voice synthesis, information identifying less than all items on the pick 
list (column4, lines 16-33). 

9. Regarding claims 64, 77 and 90, Bergman teaches wherein the formats 
consistent with both the first modality and the second modality comprise HTML and 
VXML, respectively Fig. 7, elms. 151/152; and column 7, lines 12-42). 

10. Regarding claims 68, 81 and 94, Bergman teaches further: 
determining, based on updating the inventory data, that the bin associated with the 
item is expected to be empty; prompting the user to confirm whether the bin is actually 
empty, using both the first modality and the second modality (Fig. 1 , elms. 34/36; and 
column 5, lines 25-61). 



Application/Control Number: 10/743,343 Page 7 

Art Unit: 3627 

1 1 . Regarding claims 69-70, 82-83 and 95-96, Bergman teaches wherein the first 
user input is received using a different modality than the second user input; wherein: 
the first user input is received using the voice-based modality, and 
the second user input is received using the touch-based modality (column 7, lines 43- 
51; and column 8, lines 1--2). 



Conclusion 

12. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Peterson et al . (US 70391 66) 

Barnes, Jr. (US 74871 12) 

Nagaietal. (US 6636587) 

Sravanapudi et al. (US 2001/0049603) 

Bondarenkoetal. (US 2004/0083479) 



Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to DANA AMSDELL whose telephone number is (571)270- 
5210. The examiner can normally be reached on 5/4/9. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Florian Zeender can be reached on 571-272-6790. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



/F. Ryan Zeender/ 

Supervisory Patent Examiner, Art Unit 3627 



